3 : Maximum Likelihood / Maximum Entropy Duality

In the previous lecture we defined the principle of Maximum Likelihood (ML): suppose we have random variables X1, ..., Xn form a random sample from a discrete distribution whose joint probability distribution is P (x | φ) where x = (x1, ..., xn) is a vector in the sample and φ is a parameter from some parameter space (which could be a discrete set of values — say class membership). When P (x | φ) is considered as a function of φ it is called the likelihood function. The ML principle is to select the value of φ that maximizes the likelihood function over the observations (training set) x1, ...,xm. If the observations are sampled i.i.d. (a common, not always valid, assumption), then the ML principle is to maximize: